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ABSTRACT 



This manual is designed as^a guide for the initial 
training of raters and referees in holistic evaluation of sriting 
samplesi It is intended for use by a single trainer in a group of 
twelve (nine raters, three referees) trainees^ The manual is divided 
into sections corresponding tc the procedures tc be used in the 
training sessions.. The left hand pages contain the trainer's text^ 
and include general directions and special points of interest. The 
right hand pages contain reprints from the Trainee Handbook. Basic 
information about holistic evaluation cf writing samples is provided 
in the form_ cf questions and answers. Ratings of exemplary essays are 
included . (Author/GK) 
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Purpose This manual has been designed as a guide for the initial 
training of raters and referees in holistic evaluation of writing 
samples. It is intended for use by a single trainer in a group of 
about twelve trainees (usually nine raters and three referees J. 

formal . The manual is divided into sections corresponding to the 
specific procedures that will be used in each training session. 
The left hand pages contain a text for the trainer and include both 
general directions, in lower case type like this, and reminders 
and special points of emphasis, IN UPPER CASE TYPE tlKE THIS. The 
right hand pages contain reprints from the Trainee Handbook, 
corresponding to whatever step in the training procedure is being 
undertaken. In cases where no material from the Trainee Handbook 
is being used, the righthand pages are blank. 

Time Limits . Each training session should last approximately two 
to two -and -a--half hours, depending on differences in groups. Trainers 
are urged to adhere, within reason, to these limits, since the con- 
centration required for successful training is likely to wane if 
the session is unduly extended. 
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I. EXPLAINING THE PROCESS OF HOLISTIC EVALUATION 

This section contains some basic information about holistic 
evaluation of writing samples^ in the form of questions and 
answers • 

PRESENT.THE INFORMATION TO THE TRAINEES IN THE ORDER IT IS WRITTEN 
AND DISCUSSIT.WITH THEM, ANSWERING ANY QUESTIONS THEY MIGHT HAVE, 
DON'T FEEL_DBLIGED_TO READ THE MATERIAL VERBATIM. PUT IT IN YOUR 
OWN WORDS IF YOU WISH, 

What iS- hoiisiic-ev^aluatl^n ? 

Holistic evaluation is an established procedure for rating writing 
samples. It has been used for many years by professional testing 
agencies, such as the Educational Testing Service and the College 
Entrance Examination Board. 

How does it work ? 

Each reader, or rater, reads a writing sample once, obtaining an 
impression of its overall quality. He or she then assigns a 
numerical rating to the paper, based on his or her judgment of how 
well it meets a particular set of rating criteria. 

Ho w i s- -dl f f en t -£rom-ana 1 ytical evaluation ? 

Analytical evaluation entails close attention to specific features 
of a piece of writing, which takes a good deal of time and yields a 
detailed assessment. Holistic evaluation is based on impressions of 
a piece of writing as a whole, so it occurs more quickly and produces 
a less detailed assessment. 

Are there different versions of holistic evaluation ? 

Yes^ there are several different versions, depending on the purposes 
of the evaluation. 

5 
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■ Which oiie are we going to use and why ? 



We are going to be using that version of holistic evaluation known 
as general impression marking because bur task is, ultimateiyi to 
rate a large number of essays according to their overall competence. 
General impression marking is the best holistic procedure for 
scoring a large number of essays written for a particular occasion-- 
in this case, the writing subtest of the Florida Teacher Competency 
Exam. 



Aren't general impressions bound to be^ahj^i^tive^, and therefore 
unreliable ? 

Yes, they are subjective, but when used in accordance with certain 

training principles, they can be highly reliable. 

THAT IS THE GOAL OF THE TRAINING SESSION. BY PRACTICING RATING 
SAMPtE ESSAYS AND DISCUSS IN6 RATINGS, THE GROUP WILL BE SEEKING 
TO ESTABLISH A eONVERGENCE OF SUBJECTIVE JUDGMENTS AT A HIGH LEVEL 
OF RELIABILITY. 



THE KEY FACTOR IN ACHIEVING eiSH RELIABILITY,^ OR eONSISTENCY, IS 
TRAINING^ IT IS VERY IMPORTANT, THEREFORE, THAT EVERYONE BE WILLING 
TO WORK TOWARD THE GOAL OF ATTAINING A HIGH DEGREE OF CONSISTENCY IN 
RATING ESSAYS. THIS ENTAILS FOLLOWING UNIFORM PROCEDURES, STRIVING 
TO DEVELOP LIKE RESPONSES, AND APPLYING THE RATING CRITERIA RIGOROUSLY 
TO EACH ESSAY. 

SOME ENGLISH TEACHERS MAY GIVE GREATER WEIGHT TO CERTAIN FAULTS OR 
VIRTUES IN WRITING THAN OTHER ENGLISH TEACHERS: SPELLING, PUNCTUA- 
TION, SENTENCE FRAGMENTS, PENMANSHIP, TOPIC SENTENCES, AND SO ON. 
FOR OUR PURPOSES, SUCH PERSONAL STANDARDS WILL BE INVALID AND MUST 
BE PUT ASIDE IN FAVOR OF UNIFORM IMPLEMENTATION OF THE RATING CRITERIA. 

ASK THE TRAINEES IF THEY UNDERSTAND THIS REQUIREMENT OF THE TRAINING 
AND IF THEY ARE WILLING TO ABIDE BY IT. DISCUSS THEIR REACTIONS TO 
IT, IF THERE ARE ANY. 



What scoring system will we be using ? 



The scoring system for evaluitihg essays written oh the Florida 
Teacher Competency Examination is as follows: each essay will be 
read Independently by three raters, who will assign' it a rating of 
1, 2p 3, or 4, depending on his or her judgment of its overall 
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competence. tWe will look at what these ratings mean in a minute 
when we discuss the rating criteria.) If one of the three ratings 
differs from any other by two or more, or if the three ratings are 
some combination of i 1 2, the paper will be sent to a fourth rater, 
or referee, who will read it and assign it a rating. This rating 
will then replace the most discrepant one. (We will look at this 
procedure a bit more carefully later in the training session.} 
The sum of the three ratings for each paper- -either the three 
original ones, or two originals and a referee's substitute- -is its 
final score. The range of scores for each essay is thus 3 at the 
low end to 12 at the top. 

II. INTRODUCING THE RATING CRITERIA 

PASS OUT THE TRAINEE HANDBOOKS . ASK THE TRAINEES NOT TO LOOK AHEAD 
IN THEM SINCE THE TRAINING DEPENDS ON A SEQUENTIAL READING OF THE 
MATERIAL, IN THE ORDER IT IS PRESENTED. 

TRAINER* S NOTE : IT IS CRUCIAL THAT THE TRAiNEES READ THE ANN0TA- 
TlONS TO TH£ EXEMPLARY ESSAYS ONU AETE& TJiE^JJAYE-JRATEfl -THE^SSAYS 
INDEPENDENTLY . 

Have the trainees look at page 1 in their Handbooks- - the 
Criteria for Evaluation of Essays. 

How were these criteria chosen ? 

The criteria Were developed to accommodate three specific conditions 
imposed by the writing test: 

1) that they reflect those characteristics widely accepted as 
indicative of good writing and can thus reasonably be expected to 
appear in essays written by people seeking initial Florida certifi- 
cation; 

2) that they are amenable to being '•translated*' into operational 
descriptions of levels of eompetehee that can be used by raters as 



CRITERIA FOR EVALUATION OP ESSAYS 



Rhetorical Quality 

1.1 Unity; Ah ordering and interdependence of parts 
producing a single effect: completeness. 

1.2 Focus: Concentration on the chosen topic 



1.3 Clarity: Lucidity of expression j lack of ambiffuitv 
and distortion. b y 

1.^ Sufficiency: Appropriate depth and breadth of expression 
to meet the writer's purposes and the demands of the 
particular topic. 

Structural and Mechanical Quality 

2.1 Organization: Consistent and coherent integration and 
connection of parts. 

2.2 Development: Appropriate and sufficient exposition of 
ideas; use of detail, examples^ illustrations, comparisons, 
etc. 

2.3 Paragraph and Sentence Structure: Appropriate form, 
variety, logic, relatedness of and artir.ng st ructural ^ units . 

2.4 Syntax: Appropriate ordering of words to convey intended 
meaning. 



Observance of Gonventions in Writing 

3.1 Usage: Appropriate use of language features: inflections, 
tense ^ agreement, pronouns, modifiers, vocabulary, level 

of discourse, etc. 

3.2 Spelling, Capitalization, Punctuation: Consistent practice 
of accepted forms. 
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directly as the criteria themselves; 

3) that they take into account the writing subskills identified by 
the Council on Teacher Education, 
READ AND DISCUSS THE CRITERIA 

Have the trainees look at page 2 in their Handbooks the Operational 
Descriptions of Levels of Writing Competence, 

How were these descriptions developed ? 

The descriptions are based on the rating criteria; that is, they 
use the terms which are definec in the criteria. The descriptions 
are an attempt to express as clearly and precisely as possible the 
imp ress ions a r^ter has in terms of the criteria when he or she reads 
essays of varying quality. For our purposes, there are four levels 
of quality, or competence, a rater must be concerned with. There 
could be fewer, and there could be more. Four is the best number for 
bur task because it provides enough degrees of ^ distinction to be 
both meaningful and manageable in assessing writing competence. 
READ AND DISCUSS THE OPERATIONAL DIHSCRIPTIONS , 
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ePERATiONAL DESCRIPTIONS OP LEVELS OP WRITING COMPETENCE 



4. The essay ±5 unified, sharply focussedj and distinctively 
effective. It treats_the topic clearly, completelyj and in suit- 
able.depth and breadth. It is clearly and fully organized, and it 
develops ideas with consistent appropriateness and thoroughness. 
The essay reveals ah unquestionably firm command of .paragraph and 
sentence structure. Syntactically , it is smooth and often elegant. 
Usage is unifonnly sensible, accurate, and sure. There are very 
few^ if any, errors in spelling, capitalization, and punctuation. 

3_. The essay is foeussed and unified, and it is clearly if not 
distinctively written. It gives the topic an adequate though hot 
always thorough treatment . The essay ls_well organized, and much 
of the time it develops ideas appropriately and suf f iciently • It 
shows a good grasp of paragraph and sentence structure, and its 
usage is generally accurate ahd^sensible . Syntactically, it is 
clear and reliable. There may be a few errors in spelling^ capital- 
ization, and punctuation, but they are not serious. 

2. The essay has some degree of unity and focus, but each could 
be improved. It is reasonably clear, though not invar-lably so, 
and it treats the topic with a marginal degree of sufficiency. The 
essay reflects some concern for organization and_for some develop- 
ment _ of ideas ^ but neither is necessarily consistent nor fully 
realized.. The essay reveals some sense, if hot full command, of 
paragraph and sentence structure. It is syntactically bland and, 
at timesj awkward. Usage is generally accurate, if not consis- 
tently so. There are^ome errors in spelling, capttaiizat ion, 
and punctuation that detract from the essay's effect if not from 
its sense. 

1. The essay lacks unity and focus. It is distorted and/or 
ambiguous^ and it fails.to treat the topic in sufficient depth and 
breadth. There is little orno discernible organization and only 
scant di^velopment of ideas, if any at all. The essay betrays only 
sporadically a sense of paragraph and sentence structure, and it 
is syntactically slipshod. Usage is irregular and often question- 
able or wrong. There are serious errors in spelling, capitalization 
and punctuation. 
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Have the trainees look at page 3 in their Handbooks the grid 
showing the association of the rating criteria and the essential 
competency subskills developed and mandated by the State. Note 
that all the subskills are, in fact, accounted for in the rating 
criteria and will be evaluated in the holistic rating process, 
(in other subtests of the Florida Teacher Gompetency Examination 

subskills will be tested separately rather than in the context o 

general performance. ) 

BRIEFLY READ AND DISeUSS THE GRID. 
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FIGURH 1. How the Essential Competency 
Subskills in Writing are Evaluated by a Criterion 
Guided Holistic Rating Pmcedure 



RHETORICAL STRUCTIAL CONVENTIONAL 



ESSENTIAL COMPETENCIES; Demonstrate 
the ability to write in a logical, 
easily understood style with appro- 
priate graiar and sentence structure. 



Ai Differentiate between formal and 
informal written English. 

10 fi. Hse language appropriate to the 
topic and reader. 

C. Apply basiciechanics of writing. 

S'. Apply appropriate sentence 

structure. 

E.:. Apply basic techniques for 
organization. 



F. Apply standard English usage, 
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iii. RATING THE i-XhlMPLARY ESSAYS 



llie grblip is ribw ready to undertake the central step in the training 
process- ^rating sample essays holistically . 



TELL THE TRAINEES THAT ^THEY ARE G0INS T0 PRAeTieB RATING ESSAYS THAT 
HAVE BEEN SELECTED AS GOOD EXAMPLES 0F VARi0US LEVELS OF COMPETENCE 
IN WRITING. THE ESSAYS WERE WRITTEN BY UNDERGRADUATE EDUCATION 
MAJORS UNDER TEST CONDITIONS SIMILAR TO THOSE THAT WILL BE USED ON 
THE FLORIDA TEACHER COMPETENCY EXAMINATION ; THEY HAVE BEEN RATED 
BY EXPERIENCED RATERS AND ANNOTATED IN ACCORDANCE WITH THE OPERATION- 
AL DESCRIPTIONS. AMONG THEM ARE EXAMPLES OF INCOMPETENT, AVERAGE, 
ABOVE -AVERAGE, AND EXCELLENT ESSAYS (ESSAYS RATED 1, 2, 3, AND 4 J 

Ac r^hhu^^^M^^^^ WHICH STRADDLE THE BORDERLINE BETWiiEN TWO LEVELS 
OF COMPETENCE. 

EXPLAIN THAT THE GROUP WILL BE READING* RATING^ AND DISCUSSING A 
NUMBER OF ESSAYS- -9 OR 12 DEPHNDING ON THEIR PROG RHSS AS RATERS 
DURING THE TRAINING SESSION. FOR THE FIRST THREE ESSAYS, THE 
FOLLOWING PROCEDURE WILL BE USED: 

EACH TRAINEE WILL READ AN ESSAY^ ASSIGN IT A RATING, AND THEN SHARE 
HIS OR HER RATING AND THE REASONS FOR ASSIGNING IT WITH THE GROUP. 
AFTER THE DiSCUSSiON, THE TRAINER WILL PRESENT THE RATINGS GIVEN BY 
THE EXPERIENCED RATERS AND WILL READ THE ANNOTATIONS ALOUD TO THE 
GROUP SO THAT THE TRAINEES MAY COMPARE THEIR IMPRESSIONS WITH THE 
TRAINING STANDARD. FURTHER DISCUSSION CAN THEN ENSUE IF NECESSARY. 

Have the trainees look at page 4 in their Handbooks -- Exemplary 
Essay 1. 

ASK THE TRAINEES TO READ THE ESSAY ONCE AND ASSIGN IT A RATING OF 
1, 2, 3, OR 4* MAKING SURE THAT THEY REFER TO THE OPERATIONAL 
DESCRIPTIONS. MAKE SURE THAT EACH TRAINEE DOES THE RATING IN- 
DEPENDENTLY. 

When the trainees have rated the first essay, ask each to report his 
or her rating together with the reasons for assigning it. Write 
each rating on the chalkboard as it is given. After the trainees 
have reported their ratings and discussed them in the group, report 
the ratings of the experienced raters and read aloud the annotation 
to Exemplary Essay 1, Call attention to the deviations among the 
ratings posted on the chalkboard. (in the extremely remote 
possibility that there are few or even no significant deviations, 

CCGNTINUED on page 18) 
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TOPIC I What are the essential characteristics of a good teacher? 
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, ESSAr NUMBER I 2 
TOPIC I Which person in public life do you most admire and why? 



7. 
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TOPIC I should sex education be taught in American public schools? 
Why or why not? 
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ANNOTATIONS TO THE EXEMPLARY ESSAYS 



Exemplary Essay #1 — An essay rated 2 by each of three experienced 

raters 

An essay of average competence 

Topic: What are the essential characteristics of a good 
teacher? 

Commentary: This essay treats the topic^ though rather super-. 

ficially. It reflects some concern for organiza- 
tion and deyelopment of ideas, though ±h a rudimen- 
tary way. Sentences are structurally repetitious; 
syntactically, the essay is flat . Usage Is 
generally accurate, though. the re are obvious lapses 
on occasion , particularly in pronoun references 
and agreement. There are a few minor errors ±n 
spelling and punctuation. 

Exemplary Essay #2 — An essay rated 2 by two experienced raters and 

1 by a third experienced rater 
An essay straddling the borderline between 
competence and incompetence 

Topic: Which person in public life do you most admire and why? 

Gommentary: This essay deals with the topic, but its style. 

impedes its sense. The essay lacks clarity. It 
is wordy and often syntactically awkward ^ problems 
v/hlch are compounded by cumberoOme, inflated 
usage. There are some errors in spelling and 
punctuation. 

Exemplary Essay #3 — An essay rated 1 by each of three experienced 

raters 

Ah incompetent essay 

Topic: Should sex education be taught in American public 
schools? Why or why not? 

Commentary: This essay does iittiemore_ than state, repeatedly, 
the writer* s opinion that there should be sex 
education in the schools. The topic is given 
insufficient treatment; there is virtually no 
evidence cited to support the position and no 
development of any related idea(s) .The writer 
uses paragraphs. and sentences^ but_they are 
consistently ill-conceived* The paper is replete 
with discordant usage ^ and there are numerous 
misspellings • 
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mentioh this to the group--ybu may justifiably express surprise 

in doing so-- arid go on to the second essay • ) Have the trainees 

read, rate, arid discuss Exemplary Essays 2 and 3, individually, 

in the same mariner: discuss the trainees' ratings as they are given-- 

posting them ori the chalkboard- -report the experienced raters' 

ratings^ arid read the anriotatioris aloud so the trainees can compare 

their impressions with the training standard, 

TRAINERS S NOTE : DURING THE DISCUSSION OF THE FIRST THREE EXEMPLARY 
ESSAYS. EMPHASIZE THE ii4P0RTANeE OF MAINTAINING A CONSISTENT APPROACH 
TO THE RATING OF ALL ESSAYS^ USING THE RATING CRITERIA AND THE 
OPERATIONAL DESCRIPTIONS. REMIND THE TRAINEES THAT STRONGLY HELD 
PERSONAL DISPOSITIONS TOWARD PARTICULAR WRITING FEATURES, SUCH AS 
EXCESSIVE CdNCERN FOR SPELLING, SENTENCE FRAGMENTS, OR PENMANSHIP, 
WILL BIAS ONE^S HOLISTIC IMPRESSION OF AN ESSAY. 

POINT OUT TOO THAT ESSAYS CONTAiNiNG NONSTANDARD BiALECT FEATURES 
AND/OR EXTREME STATEMENTS OF A SOCiAL, POfclTiCAL, OR PERSONAL NATURE 
MUST BE EVALUATEB--AS ALL OTHER ESSAYS- -ACCORBlNG TO THE CRiTERIA, 
REGARDLESS OF i THE RATEB.»S PERSONAL FEELINGS FOR OR AGAINST SUCH 
FEATURES_ANDSHCHlSTATEMENTS._ TRAINEES WHQ_ CANNOT OR_ARE NOT 
WILLING TO_RATE_ESSAYS IMPARTIALLY WILL NEED TO BE EXCUSED FROM 
FURTHER PARTICIPATION IN THE TESTING PROGRAM, 

The group is now ready for the second set of Exemplary Essays 

4, 5, arid 6 tpages 11-20 in the Trainee Handbook) • If the trainer 

feels that additional whole-group discussion of ratings is desirable, 

the essays may be handled in the same manner as the first three. 

If however, the trainer feels that the trainees have made progress in 

understanding and using the holistic rating process, a small-group 

approach can be adopted, as follows: 

Ask the trainees to rate the three essays independently, making sure 
to reread the operational descriptions of the four levels of writing 
competence before reading and rating the essays. Remind them to 
read each essay only orice. After everydrie has read and rated the 
essays^ break the trainees up into groups of three. (If the total 

tCOMTINUED on- page 29g3 
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number of trainees does not allow all groups to have three members, 
use groups of three and four alternately) i 



TRAINERS S NOTE : THREE IS TH1:_SMALLEST_NUMBER OF RATERS PER GROUP 
tOul^D_TO_VI£Ll5_ SUFFICIENTLY. HIGH LEVELS _OF_REL GROUPS 
QF THIS SIZE.WILL BE JSED TO RATE ESSAYS WRlTThN ON THE FLORIDA 
TEACHER COMPETENCY EXAMINATION SUBTEST IN WRITING. HENCE THAT 
NUMBER IS BEING USED IN THE TRAINING SESSIONS. 

Have each group compare and discuss the ratings each member has given 
each of the three essays. Appoint, or have each group elect, a 
spokesperson, who will be responsible for recording the substance 
of the group's discussions and for reporting it at the general 
session to follow. 

REMIND THE TRAINEES TO KEEP REFERRING TO THE OPERATIONAL DESCRIPTIONS 
IN THEIR SMALL-GROUP DISCUSSIONS. CHECK PERIODICALLY THE DISCUSSION 
IN EACH GROUP TO MAKE SURE THIS IS BEING DONE. 

After the groups have discussed the three essays, reconvene the 
trainees in general session. Ask the spokesperson in each group to 
report the members* ratings for each of the three essays, and list 
these ratings on the chalkboard by group. Then have each spokes- 
person report on the discussions that occurred in his or her group, 
noting any special problems it encountered. 

TRAINER* S NOTE : THERE WILL TYPICALLY BE MUCH MORE DISAGREEMENT 
ABOUT AND DISCUSSION OF ESSAYS OF MIDLEVEL QUALITY- -THOSE RATED 
2 AND 3--THAN OF ESSAYS OF LOW £3R HIGH QUALITY- -THOSE RATED 1 ' 
AND 4. IT JS ESPECIALLY IMPORTANT TO DISCUSS THOROUGHLY THOSE ESSAYS 
STRADDLING THE LINE BETWEEN COMPETENCE AND INCOMPETENCE- -BETWEEN 
RATINGS OF 1 AND 2. 

I£ the need arises, reiterate and discuss again the importance o£ 
adhering to the established evaluation procedures - -particularly in 
essays containing controversial features. 

After these three essays have been rated and discussed, refer to their 
annotations . 
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READ ALOUD AND DISCUSS THE ANNOTATIONS.FOR EACH.ESSAY^ NOTING HOW 
THE ESSAY FITS THE DESCRIPTION OF THE PARTICULAR LHVEL(S3 OF COMPETENCE 
IT HAS BEEN ASSIGNED BY THE EXPERIENCED RATERS. COMPARE SPECIFIC 
FEATURES OF ESSAYS TO BRING OUT DIFFERENCES IN RATINGS. 

Teii the trainees that when the actual writing subtest is administered, 

there will be two rules for referring essays receiving discrepant 

ratings to a referee, whose rating will replace the most discrepant 

one : 

IJ if any rating in the set of three differs from any other by 
two or more; and 

2) if the combination of ratings is 1 1 2. 

List all possible combinations of ratings on the chalkboard and 
underline those falling under rule #1; check that one which falls 
under rule *2, as follows: 

2 5 4 

2 4 4 

3 3 3 
3 3 4 

3 4 4 

4 4 4 



Discuss the rules briefly, if necessary. Call attention to the 
trainee ratings, listed by group on the chalkboard, of the three 
Exemplary Essays just rated and discussed. Point out the ones in 
each group which are discrepant. 
TAKE A SHORT BREAK 
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Exemplary Essay #^ — An essay rated H by each of three experienced 

raters 

An excellent essay 



Topic: What values do you feel schools should attempt to 
impart to students? 



Commentary; 



This essay- is sharply focussed, unified, and 
distinctively effective. It treats the topic 
clearly, completely, and in suitable depth and 
breadth. The essay is very well organized and 
develops ideas appropriately. It has a firm 
command of paragraph and sentence structure^ and 
it is syntactically smooth and, at times ^ elegant. 
Usage is uniformly sensible , accurate , and sure. 
There are no errors in spelling, capitalization, 
and punctuation. 

Exemplary Essay #5 — An essay rated 2 by two experienced raters and 

3 by a third experienced_ rater 
An essay straddling the borderline between 
average and above -average competence 

Topic: What are the essential characteristics of a good teacher? 



Commentary: 



This essay treats the topic adequately though not 
in much depth. it is well organized: it is, in 
fact , a classic , school-taught , five -paragraph 
theme, with ah introduetloh, a body, and a con- 
clusion. Syntactically, it is flat and repetitious 
Usage is pedestrian. The paper containr, few^ if 
any, errors in spelling, punctuation, and capital- 
ization. 

Exemplary Essay #6— An essay rated 3 by each of three experienced 

rat e r s 

An essay of above-average competence 
Topic: Which person in public life do you most admire and why? 



Commentary: 



This essay is well focussed and unified, and it is 
clearly though not distinctively written. It is 
well organized, develops ideas appropriately, and 
gives the topic sufficient treatment. Usage is 
generally accurate, and syntax ±s reliable. The 
essay shows a good grasp of sentence and paragraph 
structure. There are a few m.inor errors in 
punctuation. 



3S 



ERIC 



-32- 



Reeonvene the trainees and have them look at pages 21 -2y in their 
Handbooks- -Exemplary Essays 7, 8, and 9. Use the small-group 
approach described on page 29 in reading, rating, and discussing 
these essays. 

TRAINER'S NDTE : IF THE SMALL- GR0UP METH0D WAS USED FOR EXEMPLARY 
ES SAY S 4 ^ 5 , ASD 6 ^ AS K THE TRAINEES T0 F0RM NEW GR0UPS OF THREE IN 
WORKING WITH THESE ESSAYS, SO THAT EVERYONE MAY SHARE IMPRESSIONS 
AND RATINGS WITH AS MANY DIFFERENT RATERS AS POSSIBLE. 

After these essays have been read, rated, discussed in small groups, 

reported upon in general session, and posted on the chalkboard, 

refer the trainees to pages 40-43 in their Handbooks- - the Annotations 

to the Exemplary Essays. Read the annotations to Essays 7, 8, and 

9 aloud as the trainees follow along, and discuss them with the group 

in light of the ratings given by the trainees. 

TRAINER '^Js?BT£ ; THE ANNOTATIONS FOR ALL TWELVE EXEMPLARY ESSAYS 
APPEAR^ IN THE TRAINEE HANDBOOK, THOUGH UNTIL NOW THE TRAINEES HAVE 
NOT BEEN DIRECTED TO LOOK AT THEM. ASK THE TpftTNEES^OT TO LOOK 
AT THE ANNOTATIONS T O ES SAYS^ 10^ 11^ AnD 1^ ^ jj 4^1EY ARE TO B£ USED 
IN THfe TRAINING . SEE SECTION IV, BELOW. 



(CONTINUED on page 43) 
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TOPIC: Which gf^oh in tiublic life 
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TOPICi What are the essen.tiai ctiaracteristics--6f a good -teacher? 





^^^-tfE-^Z.^-^^^^--*^*^ .-^^2C^^<5^a^Ssdb^ ;^=:=;==:^-— 






ERIC 



^9 



36- 




<^i<^.^Zjc^^ >iJLJ^SiJ&jOi^ ^^ey 








ERIC 




-37- 



25- 









ERIC 



^2 



38- 



. 26. 








ERIC 



. . ESSAY NUI^BERi 9 

TOPiCi Haw much and in what ways does television affect you? 
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Exemplary Essay ffl—Ah essay rated 1 by each of three experienced 

raters 

Ah incompetent essay 

Topiq: Which person in public life do you most admire arid why? 

Commentary: This essay lacks. unity and focus. It wanders off 

the topic arid. fails to develop at_ all the two ideas 
preserited iri its opening lines. Paragraphing is 
virtually rionexisterit, and there are numerous 
wrorigly used sentence fragments and faulty con- 
st 2?uctioris_. The syritax is often muddled. Usage 
problems abound: inappropriate abbreviations, 
inflections, wrong words arid wrorig forms of words. 
There are many errors in spelling and capitaliza- 
tion. 

Exemplary Essay #a--Ari essay rated 3 by each of three experienced 

raters 

An essay of above average competence 

Topic: What are the essential characteristics of a good teacher? 

Commentary: This essay is reasonably v/ell fdcussed arid Unified. 

The writing is generally clear . The essay _ is ._ 
well organized arid treats the topic sufficiently. 
Usage is usually reliable, and a good command of 
sentence and paragraph structure is demonstrated. 
Syntactically, the essay is reliable if a bit flat. 
There are a few minor errors in punctuation. 

Rxeriplary Essay #9 — An essay rated 2 by tv/o experienceci raters arid 1 

by a third experienced rater 
An essay straddling the borderline between 
competence and incompetence 

Topic: How much, and in what ways, does television affect you? 

Commentary:. Except for the second paragraph, this essay does 
not treat the topic. There is some development 
of ideas which are themselves only tangentially 
related to the topic at hand. Organization is 
spotty^ though a sense of paragraph and sentence 
structu2?e is revealed. Usage is reasonably 
accurate. Syntax^ except for an occasional lapse, 
is fairly consistent . Spelling, capitalization, 
and punctuation are good. 
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IV. DETERMINING THE EFFEeTlVENESS OF THE TRAINING 



At this point, the trainer must decide whether the trainees have 

achieved an acceptably high level of rater-to-rater agreement in 

their holistic evaluation of essay samples. If so, the training 

session should be concluded according to the procedures outlined in 

Section V. If not, Exemplary Essays 10, 11, and 12 should be used 

for additional practice, using the small-group method and the 

ahhotatiohs in the same manner as for Essays 7, 8, and 9. 

TRAINER'S NQTE : AeeEPTAfifcE LEVEtS GF INTER-RATER AGREEMENT MAY, IN 
YOUR JUDGMENT, ^BE REAEHEB BEE0RE THE RATING 0F EXEMPLARY ESSAYS 
1-9 HAS OCCURRED- -AFTER THE FIRST SIX, FDR EXAMPLE^ IF THIS IS SO, 
USE ESSAYS 7,. 8, AND 9 TO CONFIRM YOUR JUDGMENT. THE TRAINING 
SESSION SHOULD NOT BE CONCLUDED BEFORE THE FIRST NINE EXEMPI:A:RY 
ESSAYS HAVE BEEN READ, RATED, DISCUSSED, AND COMPARED VVITH THEIR 
ANNOTATIONS . 



(CONTINUED on page 55) 
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•TOPlCi Wfia±_values do you feel schools chould attempt to impart to 






— 





49 



-45- 



31. 











90 



33. 

-47- 




o 

ERIC 



-48- 



3^. 



TOPIC I Should sex education be taught in American public schcjois? 
Why or why hot? 
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Exemplary Eivjay //10--An dyLsay rated 3 by two oxper i_cnec^ r^aters arid 

U by a third experienced ratet> . 
An essay straddling. the borderline .between 
above-average competence and ^jccel^^nee 

Topic: What values do you feel schools shoul<3 att^^pt to 
impart to students? 

Commentary: This essay is unified < _d well fbeuss^^. it ±s_ 
clearly and, at times^ distinctively ^'^itten. It 
treats the topic with considerable thp^'oughhess" 
given the writer's poicb of _ view->^and it ±s 
organized surely and effectively^ developing ideas 
sufficiently and appropriately. 5*1^© ^'^sa.y has 
a firm command of sentence and par^Si*^^^ structure 
it is syntactically smooth. Usage is ^ound and 
almost always accurate and sensible, ^here are 
virtually no errors in spelling^ eapi^'^lization, 
and punctuation. 

Exemplary Essay #11 — An essay rated 2 by two experienced^ i*^ters and 

3 by a third experienced rater 
An essay straddling the borderline between 
average and above-average competen<^^ 

Topic: Should sex education be taught in American Public 
schools: Why or why not? 

Commentary: This essay treats the topic adequately If not 

thoroughly. It gives some attention t^b focussing 
and shaping and has sone degree of nni-^^n It 
contains some sentence frasments, one two of 
which may be deliberate . There is sq^^ syntactic 
awkwardness J though much is clear. U^^ge is 
gene rally accurate , though some faulty P^on oun 
references are present . There are _s f*^^ errors 
in spelling and punctuation, but not ones. 

Exemplary Fssay #12 — An essay rated 2 by each of thi^ee experienced 

raters 

An essay of average competence 

Topic: Do you favor or oppose the women ^s lib^rati^^ movc^ment 
in bhe United States? Explain why. 

Commentary: This essay is a reasonably clear and r^^sonably 
sufficient treatment of the topic. It^^uffers 
at times from awkward syntax, faulty s^^tence 
structure, and "overloaded" paragi>aPhs* ^xc^pt 
for an occasional lapse, usage is fairly accurate. 
There are some errors in punctuation. 
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The trainer's decision rests on a personal judgment of the effects of 
the training on the trainees' progress as raters. To guide his or 
her judgment, the trainer can refer to the following expectations, 
which experience has shown to issue from a well-managed training 
session involving serious, co-operative trainees: 

IJ the ratings of essays done after some practice and discussion 
have taken place should show greater inter-rater agreement than those 
done at the beginning of the training session; 

23 as the training proceeds, discussion about ratings should center 
more pointedly on specific criterial elements and their execution in 
a particular essay; 

3) raters should show an increasing willingness to use the operation- 
al descriptions of writing competence in rating essays and to cite 
relevant elements of them in discussion. 

As ah objective measure of progress, the trainer can keep a record 
of the number of essays receiving discrepant ratings [those which 
would be sent to a referee). The number of these should decline 
as the session continues and the trainees become more accurate and 
reliable raters. While it is difficult to predict the actual number 
of essays v/hich will receive discrepant ratings in a training 
session, it is not unusual for up to 50% of those rated during a 
first attempt at holistic evaluation to receive discrepant ratings. 
Similarly, a drop in discrepant ratings to about 20% or 25% is not 
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uricbmmbn for essays rated hear the end of a traihihg session, after 

considerable practice ahd discussibh have taken place. 

TRAINER'S NOTE : IN CONCLUDING A TRAINING SESSION^ CALL TO THE 
TRAINEES' ATTENTION AS MANY OF THESE INDICATORS AS HAVE BEEN APPARENT 

If the trainer feels that additional training is warranted beyond 

that provided by the twelve Exemplary Essays, another rating session 

can be conducted using the three backup essays included in the 

Trainee Handbook (pages 57-66 in this manual). 

IT IS EXPECTED THAT NO ADDITIONAL TRAINING WILL BE NECESSARY. THE 
BACKUP ESSAYS ARE PROVIDED MAINLY IN ANTICIPATION OF ANY SPECIFIC 
PROBLEMS THAT, IN THE TRAINER'S OPINION, CAN BE ELIMINATED BY ONE 
MORE BRIEF RATING SESSION. 

(CONTINUED on page 67) 
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TOPJCi; Shbtt3;d sex education be taught in American jsublic schools? 
Why or why hot? 




JL^j^^jui. 'Jo mod/v^ '4^Vd\ip iSk/aA ^-k^JLk^ 
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' TOPIC I Do you favor or oppose . the women's liberation movement in 
the United states? Explain why. 



^7. 






L^4. .h^e.-rmi5L2 

iiiCDme^ ^od^. - - 






ERIC 



fsu&i&^ inA/oiAJeM "nam. tziAen 

-f^gj ^Ao^ 
..U>^Dmer\s uJoeAa:}!^ mo^/emerc^ ^ 




-61- - 




•62- 







-t^yuA OiAJTL GUrxal^ 



7 







-63- 



50. 




f78 

ERIC 



V -6A-- , 

- - - - - -- - - - - --- - - -- - -- - - - 

TOPjCt yfidch person in public life do you most admire and why? 

^yi£cudlju, mAi. ht^-^^ rnouh- LOkjQ _£4 

Jj/loALd _tu^ pUif^f-fStxA^A^ i-t^^ 

i;t5 .QJUL^ =-^^^^^ -ctd~<3- GCukplTbte. ^ 




i^O^/, vVUln>.Lci - Ic-^AA- hcui -'^- -^-^ -4-^x^1 - 



!:co/u-6A . ca as fejutat -^otn ^outi, ./u^c cofix^vOKu^-^ - - 



i.^diair-O^ -ApouvLouLoU, /ko/^ti^ a-*vot . cEMuooa^sI* 
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V. CONCLUDING THE TRAINING SESSION 

Have the trainees look at page 54 in their Handbooks the Guidelines 
for Raters. 



R1:AD AND DISCUSS THE GUIDELINES, .REMIND THE.TRAINFES TO KEEP THEM 
HANDY AND TO REF^R TO THEM JUST BEFORE RATING A SET OF ESSAYS, 



Explain the schedule that the trainees will follow during the time 
they are evaluating essays. Assign the trainees to the work stations 
they will b:: using during the evaluation sessions. Answer any final 
questiOii> ;:hat arise. Dismiss the training session, giving the raters 
a tiiuj t report to their stations to begin work. 
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GtJIDELiNES FOR RATERS 



Before beginning to rate a set of essays^ remember to- 

1. Reread the four operational descriptions of levels of 
competence. 

2. Establish a mental "set" so you are ready to cof^^entrate 
oh reading and rating the set of essays, wltJ^ou* dis- 
traction, from start to finish. 



While reading a set of essays^ remembei? to: 

1. Read and rate the essays in the exact order you ^^ceive 
them, 

2. Approach each essay separately; don't compart ot^^ With 
another. 

3» Read each essay only once^ unless the quality of ^he 
penmanship or waning concentration demands a ^^ei'^^ding. 

Avoid being influenced by your owi_ disposit i^^ns '''pward 
particular writing features (spelling, purictuati'^^^ 
sentence fragments, topic sentences, penmansliip, ^tc.)» 

5. Avoid being unduly influenced by nonstanclard _ ^i^^^^t feat 
and by extreme statements of a social, pollti^ait* or 
personal nature. 

6. Stop after every ten papers or so and take a shc7^t mental 
breaicj relax? t'S<.e a deep breath, etc., befor^ r^^^uming 

. to the rating task. 



Referees ' should make certain to: 

1. Read the right essay. 

2. Return the essay to the exact place in the set comes 
from. 

3* Record the rating next to the proper code hummer ^n the 
Rater* s Tally Sheet. 



